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The optimum signal and receiving filters for a data transmission system 
with a fixed channel and detection process are found. The criteria of opti- 
mality are: (i) the minimization of mean square error between the input 
multilevel data signal and the output to the decision threshold and (ii) the 
minimization of noise power output with no inter symbol interference. 
Explicit frequency characteristics are obtained for the separate and joint 
signal and receiver optimization problems. For the joint problem it is seen 
that some freedom exists in assigning phases to the transmitter and receiver. 
In addition, explicit equations are obtained for the output signal-to-noise 
ratio for the problems considered. The optimization procedures are carried 
out in detail in some examples. In comparing binary and correlative multi- 
level signalling (e.g., duobinary), it is seen that there may be channels for 
which the optimum multilevel system is superior to the optimum binary 
system. 

I. INTRODUCTION 

In most instances a data system designer is faced with the problem of 
transmitting through a noisy channel over which he has no control. 
In other words, the designer must concern himself with transmitter and 
receiver terminals which reduce the disturbing effects of the channel. 
Nyquist's classic paper 1 considered over-all system designs which elimi- 
nate intersymbol interference. Others (see, for example, Ref. 2) have 
also reduced the effects of noise by designing the transmitter and 
receiver to minimize the noise output for a given Nyquist characteristic 
and an ideal channel. Tufts' recent work 3 " 6 has recognized that trans- 
mission without intersymbol interference may not be the most desirable. 
He has considered the problem of optimizing transmitted signal wave- 
forms or receiver filters under the criterion of minimizing the mean 
square error (thus minimizing the joint contribution of noise and inter- 
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symbol interference). His attempt at joint optimization of the trans- 
mitter and receiver was successful only with the added condition that 
the transmitted waveform be time limited to one bit interval. This is an 
important case, but the unsolved joint optimization problem without 
this constraint is also important. The joint optimization solution would, 
in effect, provide a performance bound for a given channel. 

This paper extends Tufts' results by solving the joint optimization 
problem (mean square error criterion) for a time-invariant transmitter 
and receiver subject only to an average power constraint on the trans- 
mitted waveform. Explicit equations are obtained for the signal, re- 
ceiving filter, and the output signal-to-noise ratio for both the separate 
and joint optimization. In addition, the optimum signal and receiver 
are found for an arbitrary channel subject to the constraint of no inter- 
symbol interference. The output signal-to-noise ratio for this case is 
obtained and is found to be only slightly less than the optimum for 
usual noise levels. The results show clearly the importance of the ratio 
of the noise spectral density to the square of the channel amplitude 
characteristic on the design techniques and the performance bound. 
The phase characteristics of the channel are irrelevant to the bound 
and there is some freedom in assigning phase characteristics to the trans- 
mitter and receiver in the joint optimization problems. 

An important reason for the solution of the joint optimization problem 
is in the notation used. The frequency domain is broken into disjoint 
intervals and the characteristic within each interval is considered as a 
separate function. The equations to be minimized are easily stated and 
the constraints (such as no intersymbol interference) are compactly 
and completely represented by this notation. Finally, the joint solution 
is easily obtained with the notation used here whereas with the standard 
approaches, 6 the solution is obscured. 

II. GENERAL CONSIDERATIONS 

Fig. 1 illustrates the general linear, time-invariant, noisy (zero mean) 
multilevel transmission system considered. One may assume that the 
information is contained in a random sequence of impulses (of weight 

ai , a t = { -2M, • • • , 0, • • • 2M - 2, 2M) 

or 

{-2M - 1, ... , -1, 1, ••• 2M + 1} 

and spaced T seconds apart) at the input of the system. Thus, a signal, 
s(t), having an amplitude of a t is transmitted every T seconds. The 
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Fig. 1 — General digital transmission system. 

noiseless system output, r(t), has the Fourier transform 

R(u) = S(w)T(a)E(<a) 
A sequence of input signals 

S ojs« - it) 



(all sums from — <x> to w unless otherwise noted) will, in general, 
produce a sequence of overlapping output pulses 

£ oif (* - IT). 

i 

The total output at the sampling time (taken at t = without loss of 
generality) is 



v = oor(0) + E aH-lT) + n(0) 



[2a) 



or 



v = a r + 23 a * ? '-* + ^o 



(2b) 



/^o 



where r* , / 9^ represents the intersymbol interference and r is the 
output value at the main sample point. 

In order to assess the performance of such a data system, it is neces- 
sary to choose a criterion of quality. One useful measure which Tufts 3 " 6 
has used extensively is the mean square error. This paper considers a 
slightly different criterion by using a normalized mean square error 

2? {[normalized input sample — output sample] 2 ) 
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or 

E{[a r — (a r + 2 a i r -i + ™o)] 2 } 

1*0 

where a simple change of amplitude between input and output has been 
eliminated as a factor. Then 



M.S.E. = E{[a r - (a r + X) atr-i + n )] 2 } 



1*0 



where 



= E{n 2 } + £{2woZ air- t } + £{ (£ cur^) 2 } (3a) 

= E{n 2 } + E{ 2 <W_*r-,,} (3b) 

= o- 2 +a/ 2 n 2 + 2 2 aj-ay+m £ nn+m (3c) 

J?*0 m=l 1*0,— m 



a* = E{n<?}. 



This expression cannot be an absolute measure of quality but should 
be related to the output level r . Much of the remainder of the paper 
will be concerned with the design of the transmitted signal and the 
receiver filter (both separately and jointly) to minimize the mean 
square error given by (3c) for a given value of r . In addition, this 
expression will also be minimized subject to the constraint that the 
intersymbol interference be zero, i.e., 

2>, 2 = o. 

1*0 

It is particularly useful for our purposes to work with the frequency 
characteristics of the system. Appendix A gives the details of the trans- 
formation of the mean square error expression from a time domain to a 
frequency domain function. In terms of the system component trans- 
forms 

Transmitted signal S(ui) = A(cS)e 3aw) 

Channel characteristic T(a>) = B(u>)e JI3M 

Receiver filter E{u) = C(a ) )e i7(u ' ) 

Noise spectral density = <£(a>) 

and the shorthand notation A[u + (2nir/T)] = A n (u) etc., the mean 
square error may be written 
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M.S.E.- (' fec'W*^) 

J-irjT { n 

+ % Z A n (u)B n (u)C n (u) cos k(«) + /3n(w) 

J. _ n 

+ 7. («)1 " S^fo) (4) 

+ y [Z A.(tt)B.(t*)C li (ii) sin k(«) + 18.(1*) 
+ 7- (M)]JFU)|dii 



where 



f(ti)=o/+2j] aja j+m cos umT. (5) 

m=l 

In (4) the frequency range has been split into disjoint bands of width 
2ir/T and the characteristic within each band is treated as a separate 
function {e.g., A»(u) = A[u + (2mc/Tj\ is A(w) for (2n - l)ir/T < 
co < (2n + l)r/r}. For simplicity, the argument, w, will not be made 
explicit (except for the dependence of the Lagrange multipliers) through- 
out the rest of the development. 

The last two terms on the right side of (4) represent the intersymbol 
distortion. By simultaneously making these terms zero, one has the 
necessary conditions for transmission without intersymbol interference 

Z A n B n C n COS («„ + fin + y.) = ~ (6) 

n ^T 

£ AnBnCn sill («„ + 0» + 7 ») =0 

(7) 

- - < 1* < - 
T ~ ~ T 

which have been discussed in a previous paper. 

In the remainder of the paper, the optimum signal and equalizer 
(receiver filter) will be determined (separately in Section III and jointly 
in Section IV) under the constraint that the average signal power is 
limited and the output level is fixed. That is, the mean square error 

[^ JZ C7n 2 #« + ™ (Z AnBnC n COS («. + A. + 7,.) " ^J 

+ ^ (? A » B " C » sin (a " + *• + *»))/ dw 
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will be minimized subject to 

/ 2 A n B n C» cos (a„ + /3„ + 7„) dw = r (8) 

J-tt/T n 

and the average power constraint which may be written 8 (assuming 

oj-o) 

P = f F(a>)A 2 (w)dco = [ FE A n 2 dM. (9) 

J_oo J—r/T n 

Note that this differs from Tufts' development 5 in that he considers a 
fixed pulse energy as the constraint. 

In addition, the suboptimum signal and equalizer which eliminate 
intersyinbol interference will be found by minimizing 

/ 22 C»*» du 

J-t/T n 

subject to the constraints given by (6), (7), and (9). 
Note that (6) contains the constraint on r , given by (8) . 

III. RESULTS OF THE SEPARATE OPTIMIZATION OF SIGNAL AND RECEIVER 
FILTER 

3.1 Simultaneous Reduction of Noise and Intersymbol Distortion 

Appendix B shows the details of the minimization of the M.S.E. 
given by (4) subject to the constraints of (8) and (9). For a specified 
receiver filter the optimum signal characteristics are given by 

sin («* + k + 7*) = (10) 

and 

T(2r - \j/F) B k C k 

k= 4 * ~E*W + 5£ (11) 

k 6TT 

where it is assumed that F ^ at any point. The constants Xi and X 2 
may be determined by using the constraining equations. 

For a fixed signal, the optimum receiver characteristics are given by 

sin (a* + k + 7*) - (10) 



and 



T(2r - Xa/F) AkB k F/$ k 

k $>k 2t 
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The coefficient is found in Appendix B to be 

T(2r - \ 2 /F) 
4tt 



r T 


r' T ? * t du i r" 2, du 


2tt 


k <Pt 2ir h $k 2w_ 



(12b) 



L 



r/T 



k <Pa 



'-T/T 



FT, 



AuB? 



T 

2t 



The ratio of the signal output to the square root of the mean square 
error is 



VM.S.E. 



2tt 

T 



r /T Fdu r' 

J-ir/T A J-xl 



V AkBk , 

A 2w \J-r/T A 



du 



J- T /T A 



(111 



(13a) 



where 



k <Pa- 2ir 



This function may be approximated by 



Ai 



2tt 



Vm.s.e. ~ r 



'1 



T/T 



(/l/. 



A/B 



'-x/T y^ _ 



- _ i r /r 

t 2x J—tc/T 



du 



k *fc 



13b) 



for large transmitted power to noise power ratios 

i k «Pfc 

In each case, the variable phase should be adjusted to produce zero 
phase (with any fixed delay removed) across the band 

(i.e., a l: = -/3 fr - y k ). 

The amplitude characteristics may be considered to be the product 
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of a term which depends upon u and k and one that only depends upon 
u, e.g., 

CM = [B k (u)A k (u)/Mu)]q(u) (14) 

where 

/ , 2/r,7r\ / , 2Ztt 

9 \ + ~T) = q \ w ~Y 

for all k, I. The function q(u) may then be considered as the output of a 
tapped delay line 

p 

since this function satisfies the condition on q(u). Thus, the solutions 
may be interpreted as linear filters matched to the signal and channel 
followed by a tapped delay line, e.g., 

C(«) = [B(»)A («)/#(«)] «(«). (15) 

3.2 Minimization of Noise Output Power with No Intersymbol Distortion 
The minimization of 

/K/T 
£ C„ 2 *„ dw + X 8 (m) £ ^n-B n C„ cos (of» + /3„ + 7«) 
7r/r n n 

+ A 4 (w) £ 4n£ n C„ sin (^ + ft, + 7 ») + Xi / F E ^„ 2 efo 



subject to the constraints of (6, 7, 9) is similar to the previous case and 
will not be presented in detail. For a fixed receiver filter, the optimum 
signal characteristics are given by 

sin U + fiu + 7*) = (10) 

and 

A k = =^-B k C k . (16a) 

The coefficient is easily found and 

. r Q T B k C k 

^"STEaW (16b) 

k 

For a fixed signal, the optimum receiver characteristics are given by 
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sin (a k + k + 7<t) = 



and 



C k = 



-X 3 (m) A k B k 



2 $* 

Again, the coefficient is easily found and 



C t = ^ 



$/.• 



E 



A k -B k - 



The ratio of signal output to the noise output is 



To 


2tt 
T 


' r " T du 


V«>' 


J-t/t y^ -At -B/t 



(10) 



(17a) 



(17b) 



(is; 



This function is seen to be identical to the first term of (13b). The differ- 
ence between (18) and (13b) is small under the condition that 

2 



2 ^E 

1 k 



A k 'B k ' 
$k 



» 1 



and one does not sacrifice much in designing for no intersymbol inter- 
ference. In this case, there is no significant advantage in using the true 
optimum design with large signal-to-noise ratio systems. 

The interpretation of the above results is the same as for the previous 
minimization problem. 



IV. JOINT OPTIMIZATION OF SIGNAL AND RECEIVER FILTER 

4.1 Simultaneous Reduction of Noise and Intersymbol Interference 

The over-all optimization requires the minimization of (4) with re- 
spect to both the signal and receiver functions A k , C k , ak and y k . 
From the minimization details in Appendix B, this requires the solution 
of 



Ak = 



sin (a k + fi k + 7*) = for all k, 



(10) 

(19) 



and 
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ft . -* M? j ? 4AC . _ r ^ _ x_ 5 ) j fm . all k (20) 

where it is assumed F ^ 0. In some correlation schemes F may be zero 
at certain points and it is easily seen that A k = C k = at these points. 
For (19) and (20) to have nontrivial solutions it is necessary that the 
determinant 

-GT£{?«*-£(— SF 

vanish. This requires 

?^"-s(*°-f) = ± 2 ™ f °" allfc (21) 

In general (unless <£>**/#*.• is a constant with /c) for any u the equation 
can only be satisfied for one k. This means that for any u there is only 
one value of k for which there is a nontrivial solution and for the other 
values of k the solutions vanish. 

The question now becomes one of choosing, for each u, the proper 
Ak and C* which leads to the true minimization. Combining (19) and 
(20) 

A k = Mi C k (22) 



and using 



one gets 



Xi*F* 



M -£(» -$-£»£ 



Ak =rA 2r °-F-imj^B k ¥> (23) 

Equations (23) and (24) represent the nonzero solutions. The best k 
(frequency region) for any u is the one which minimizes the function 

['[ [E C k *(u)* k (u) + ^ |E A t («)ft(«)C t (tt) - ^} 2 ] dw 

where the dependence on w has been reinserted for clarity. For every u 
there is at most one C k and A k with a nonzero value so the function 
may be written 
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E f L [c* 2 (iO**(«) + ^F(u)L k (u)B k (u)C k (u) - ^V] du 

where L k is the subinterval within the interval 

[(2/c - 1) (t/T), (2k + 1) (t/T)] 

over which A k (u) and C k (u) have a value. It is necessary for the true 
optimum that L k be chosen to minimize the above expression. The true 
minimization may not require the use of the total interval — t/T < u < 
t/T, i.e., it may be possible to decrease noise output faster than inter- 
symbol distortion is increased by using a smaller interval. Because this 
situation only arises when $/.V#a- in the optimum L k regions have large 
variation, and is cumbersome to consider in detail, it will be assumed 
that nonzero A k and C k will exist over — t/T < u < t/T (unless 
F — 0). Using (21, 23, 24) the above function may be written 



?/jf; 



X, 2X x W(w) \ AiW(u)F J (w) 



it LM **(«) B k (u)F*(u)) B k (u) 

T y («) / x » 2\ 1 W(m) \ 2 "| 

+ 4* 2 \ F(u) B k (u)F*(u)j J 

The constants may be evaluated by using (21, 23, 24) and the con- 
straining equations. Thus, 

n = j_ Z A k B k C k du (8) 

since the total range of all L k is the interval on u ( — t/T, t/T) and 

Using the average i)Ower condition 

F= /"' fEV& (9) 

J- t/T k 



-$UM- 



J5 t F* I X»fi* 



(27a) 
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27rXi» k h k B k ^ 2tt V J i 



L k B k 2 



(III 



+ 



n 

4ttX 






(27b) 



dw. 



Combining (27b) and (26) one gets 



27T k ll k D k 



p + 



!tt V •//,* 5* 2 



(/// 






& •'it 



(28) 



Finally, combining (21-23, 25 and 27) 

'f E/ *J^* 



M.S.tf. = = 



i \n 



2t k J L k B k 2 2,t 



■(?/*£»*) 






(29) 



Equation (29) represents the mean square error in the transmission 
and Lk must be chosen to minimize this expression. This is rather 
complicated but to a close approximation, the minimization of 

k J L k Dk 

corresponds to the minimization of (29). The minimum mean square 
error results by choosing for each value of u 



the k for which 



7T T 

— — < W < — 
rp = M = rp 



***(«)F*(t«) 



Bk(u) 
is a minimum. The interval L k is that region of u for which 
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B k (u) 

is a smaller than any other 

flMFHu) 

Bj(u) • 

Since F(u) is independent of j, L k may also be defined as the region 
for which $**(tt)/B*(tt) is smaller than any other $j*(u)/Bj(u). Thus, 
the choice of L fc to minimize 



* J /. fc />* 



J 
e2u 



also corresponds to the minimization of the M.S.E. and the regions in 
which power is transmitte d are independent of the correlation properties 
of the data. 

Once the L k are chosen, the resulting characteristics are 

Ak '~p+^I &*-£(?/ A*)'/?/ T 

■67T fe J t A . «T Z7T \ A.- J L k Diet* / I k J L k t 



?/ 

* J In 






(30) 



X ^F3 + 



£?/,^ r/ V?/ £4 y] 



**' r ** 



fi*/N 2tt B*»F 



and 



2 \iF 



<V = ^- V (22) 

where Xi is given by (28). For (30) to be a solution requires that A k ^ 
or 

**' < ft + $ (31) 



for the regions defined by the L k . Taking into account the difference 
in problems considered, this equation corresponds to Tuft's condition 
for a solution [Ref. 5, (28)]. Equation (31) states that the solution 
must be limited to regions without sharp peaks in the $> k \/B k F^ char- 
acteristic. The phases, a k and 7a- , may be arbitrarily assigned to A k 
and Ck subject to the condition of (10). The ratio of the signal to the 
square root of the mean square distortion is 
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r 2tt 

VM.S.E. = T 

v I 2t k JL k B k 2 2w \ k •>L k B k t 1 ' ) i k JL k t J . v 

E/^ 



dw 



'** 



It is difficult to make a general comparison of this result and (13) be- 
cause of the arbitrary signal, A k , which appears in (13). A comparison 
for a specific case is found in the examples of Section V. 

4.2 Minimization of Noise Output with No Inter symbol Interference 

From the previous results, the minimization of the noise output with 
respect to both the signal and receiver functions, A k , C k , <x k and y k 
requires the simultaneous solution of 



sin (a* + Q k + 7*) - for all k, (10) 

2XjF 



A k = =^-B k C k (16a) 



and 



-TMAjJh fora]]/b (17a - 



Ck — ., , 

2 <S> fc 

where again nonzero F is assumed (if F = 0, C k = 0, A k = 00). A non- 
trivial solution of (16a) and (17a) requires that the determinant 

_ Xl(u) Bl 
4XjF $* 

vanish. This requires 

Uu) = ± 2 2±*W. (33) 

B k 

Again, for a particular u, the right hand side of the equation depends 
upon k and can only be satisfied for one value of A;. For the other values 
of A;, (16a) and (17a) have trivial solutions. At each value of u there 
must be at least one nontrivial A k and C k because of the condition 

EiiAc. -g -|is«a|. (6) 
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Thus, for each value of u there is one and only one nonzero An and Ck 
and the question becomes one of finding which k leads to the true 
optimum. 

Using the same techniques as shown in the previous section, the noise 
output is given by 

E/ *•<.*(.)*.- (£7 i(E/ ^4 04) 

k •'/.* \2ir/ I { k JL k Bk{u) J 

This expression is minimized by choosing for each value of u, the k 
for which 



B k (u) \ B k (u) 

is a minimum. The interval L k is that region of u for which 

* k Hu) 



is smaller than any other 



Bk(u) 



The resulting characteristics are easily found to be 



and 



*-4?/*tt*Joti (35) 



Note here and in the previous case, the bound depends upon the ratio 
$fc*/J3* but the transmitting and receiving niters required to achieve 
the bound depend upon <p* and B k separately. 

Again, the phases, a k and y k , may be arbitrarily assigned to the 
transmitter and receiver subject to the condition of (10). The ratio of 
the signal output to the noise output is 

Comparing this equation and (32) shows the degradation in the per- 
formance bound to be expected when no intersymbol interference is 
allowed. This degradation is small under the usual condition of low 
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noise. The two equations are identical if 



B k {u) FKu) 

If this condition holds, the optimum solution is one with no intersymbol 
interference. This condition is especially useful with uncorrelated data 
where F is a constant. 

v. EXAMPLES 

Consider the problem of transmitting uncorrelated binary data 
(a i = — 1 or 1, and F = 1) at a rate of 1/T bits per second over the 
channel whose amplitude characteristics B(u) and noise spectral density 
function $(u>) are shown in Fig. 2. For ease of presentation these func- 
tions are limited to the region (—2ir/T, 2r/T) and the phase character- 
istic is assumed to be zero. The preceding discussion will be illustrated 
by assuming a fixed signal and then finding the optimum receiver filter. 
The bound for the utilization of this channel will then be found by 
finding the joint optimum receiver and signal combination. For this 
purpose, the function 



B(<0) = C05^ 



$(oj) = Ncos 



wJ 



* ,/2 (w)/BM 




Fig. 2— (a) Amplitude characteristic; (b) noise spectral density; (c) optimum 
operating region. 
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B(«) 

is given in Fig. 2(c). 

The various quantities of interest are given in the figure and it will 
be assumed that the initial choice of a signal gives a cosine roll-off and 
produces no intersymbol interference 



. , v , /PT ioT 

AM = V 27 C0S T 



(38) 



where the signal normalized to the average power restriction. The 
initial receiver filter is assumed to be flat. For this system 

• 2r/T 

2 Wl 

Id) 

/ l->~ /P 

(39) 



/•i) 



Vc 



2 Wi , 

cos — — aio ,=— i — 

_ ui* 4 /P, _ fP 

\r' T ^7? v± N ~ Vn 

/ cos — — du 



For no intersymbol interference, the optimum receiver filter is given by 
A k B h 






2tt 



** 



ITS 1 



** 



5* 



3 wT . 3 uT 

cos — sin -— 

4 4 



c* 



" 2tt y pt 



B* 



3 mT , . 3 mT 
cos — — h sin — — 
4 4 



- y ^ u^O 



O^u^f 



(17b) 



(40a) 



(40b) 



and 



A] 



Vc 



2tt 






du 



£ 



2 R 2 



A*»B. 



T ATT /o 



r/T 



c2u 



~H 



2ttP r2.47T 
NT T 



3 uT . . 3 uT 
cos — — |- sin — — 



(18) 

(41a) 
(41b) 
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For the optimum filter where intersymbol interference is allowed, 



C\ = 



L 



3 uT . . 3 uT 
r/T cos — - + sin — du 

3 uT , . 3 uT ,N 

cos — + sin -— + -j3 

4 4 / 



3 uT . . 3 uT . N 
cos T + sin — + j 



(42) 



^ u ^ f 

which will not be much different from the result with no intersymbol 
interference for low noise. The optimum detector with no intersymbol 
interference (40a,b) is shown on Fig. 3(a). 

The joint optimization requires constraining C(co) and A(ui) to the 
region — r/T ^ u ^ r/T. From Fig. 2(c) this is seen to be the region 
for which 

<t>'(o>) 

BW) 



C(w) 




A(o) 



C((u) 



(b) 



(C) 




27T 
T 



T 



21T 
T 



Kg. 3 — (a) Optimum receiver shaping for given A (w); (b) signal shaping for 
joint, optimization ; (c) receiver shaping for joint optimization. 
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is a minimum. The optimum signal and receiver for no intersymbol 
interference are, using (35, 36), 



An = 



Pi 



1 



"' T du 



iuT 

m, COS — 

cos 



(43) 



C a = 




2ttP5 



cos* — - 
4 



(44) 



and are shown in Figs. 3(b) and 3(c). The resulting ratio of signal-to- 
noise is from (37) 



n 



2w P[ 

T M 



L 



" T du 



■ it k uT 
cos — - 
4 



0.95 A/lz 



(45a) 



(45b) 



The time optimum filters, with intersymbol interference, will be close 
to the previous results for low noise. The improvement in signal-to- 
noise ratio may be obtained from (45b) by replacing P by 



P + 



r 



N 



T 



2tt i-rlT uT 
cos -— 
4 




which is greater than P. 

As a further illustration, consider the utilization of the channel whose 
amplitude characteristic B{u>) and noise spectral density function $(co) 
are shown in Fig. 4. The main point of this example is that the solution 
of the joint optimization problem is not confined to the interval 
{ — it IT, w/T). It is, as a matter of fact, split up into three regions; 
L\ for —t/T ^ u < —a, L for —a < u < a, andL_i for a < u ^ v/T 
as shown in Fig. 4(c). These turn out to be the regions of minimum 

B h (u) 

as can be seen by the curve Fig. 4(c). The optimum receiver and signal 
are given in Fig. 5 for the case of no intersymbol interference. 
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B(w)=COS^I 



*(w) = Ncos 2 ^ x 



(c 



.-1/2 (1>T 



^I + cos^?) 



$' /2 (w3/B(w) 




Fig. 4 — (a) Amplitude characteristic; (b) noise spectral density; (c) optimum 
operating region. 



The previous two examples have considered the problem of finding 
the optimum regions, L k . , for transmitting energy. Once these regions 
have been determined it is of interest to examine the effect of F(u) 
(the one parameter over which the designer may exercise some control) 
on the performance bound. In particular, since it is generally considered 
that a binary system is optimum for transmitting data in fixed time 
slots, it is interesting to ask if some sort of correlation scheme leads 
to a better output signal-to-noise ratio than binary for the same average 
input power. In other words, using (37) [the case for no intersymbol 
interference] are there cases in which 



2/ 

k J L k 



F Ku)* k h (u) 
B k {u) 



du 



i- J i 






du? 



(46) 



It should be emphasized that the signal-to-noise ratio is being exam- 
ined and not the error rate. For correlative multilevel schemes which 
lead to a solution of (4G) the error rate may be greater than or less than 
that of the binary system. In addition, there are certain error detecting 
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A(cu) 



CM 




Fig. 5 — (a) Signal .shaping for joint optimization; (b) receiver shaping for joint 
optimization. 

features of the correlative multilevel systems'" which make it difficult 
to compare its error rate to binary. 
Equation (46) will be easier to consider if it is written 



w " F { (u)^(u) 



rrlT 
•l—r/T 



(III 



*f 

J-tI 



**(m) 



du 



(47) 



B(u) " ~ J-t,tB{u) 

where <f>(w) and B(u) are the composites of $*(u) and B k (u) in the (gen- 
erally) disjoint regions L k in u>. The total interval over which the Lie's 
extend is —ir/T < u < w/T. For the case shown in Fig. 4 one would 
have 

r#i*(i*)/ft(«) 

i«tt*(i*)/B (u) 
^(tO/B-iCw) 



B(ii) 



-v/T < u g -a, 
— a ^ w ^ a, 
a ^ W < tt/T. 



As a specific example, consider correlated data which is formed by 
adding binary ( — 1, 1) bits separated by T seconds (i.e., adding con- 
tinuously the present bit and the immediately preceding bit). Such data 
(duobinary) has three levels ( — 2, 0, 2) and 

F(u) = 2 + 2 cos uT. 
It is easily shown that if 
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**(t*) , , uT 
= do — di cos — - 



B(u) ^ 2 



where 



do ^ di ^ I — t do « 0.75 do 

(47) is satisfied. This means that for the above characteristic the opti- 
mum duobinary transmission system operates with greater output S/N 
than the best binary system. It is obvious that for 

B(u) 

constant, (the usual case considered) binary transmission is optimum 
(with duobinary poorer by a factor of ir/4 as pointed out by Bennett 11 ). 
This example is not intended to exhaust the possibilities but to point 
out that for deviations from flat channels and white noise there is the 
possibility that signaling with correlated multilevel data is superior to 
binary. 

VI. CONCLUDING REMARKS 

The joint receiver and signal optimization has been carried out 
for the general data transmission system. The optimization considers 
simultaneously the noise and intersymbol distortion. The results may 
be applied to bandpass (baseband transmission, not carrier) or gradual 
cutoff systems as well as the sharp cutoff systems illustrated in the 
examples. Unfortunately, the joint solution is unrealizable, but it does 
provide a bound on the transmission performance with a fixed channel. 
Thus, with a given channel, one can do no better than (32) for the ratio 
of signal mean square error, and no better than (37) for the output 
signal-to-noise ratio with no intersymbol interference. From these 
equations, the importance of the ratio 

is clearly seen. It is the only contribution to the bound other than the 
signal correlation function F, and the average signal power. Notice that 
the phase characteristic of the channel is irrelevant to the bound. 

In the first example, it is seen that the original choice of signal and 
receiver was close to the optimum bound. Hence, there is little improve- 
ment to be gained with more complex processing. The second example 
illustrates a case where a more complex signal and receiver would be 
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theoretically useful. The signal and receiver of the first example would 
not effectively combat the noise in this case. The last example illustrates 
an advantage of correlative multilevel signaling in matching certain 
channel characteristics. 

In the examples, the suboptimum (no intersymbol interference) 
solutions arc presented in detail for three reasons: 

(i) The solutions are of simpler form but just as illustrative as the 
optimum solutions. 

(it) The solutions depend only on the shape of B(w) and <p(co) and 
not upon the relative noise and signal powers. 

(Hi) The optimum and suboptimum solutions are nearly identical 
in the standard situation of large signal-to-noise power. 

The explicit results for the suboptimum case are possible because of 
the explicit and complete statement of conditions for no intersymbol 
interference [Equations (6) and (7)]. 

APPENDIX A 

Frequency Domain Representation of Mean Square Error 

By writing the time domain samples, n , in terms of the Fourier 
transform of r(t) one gets 

r(t) = [ R(a)e jul du (48a) 

if qQ 

n = f B(«y' wIr du (48b) 

J— oo 

oo /.(ir/r)(2n+l) 

r, = E / R(o>)e''° lT da, (48c) 

„ oo J(t/7')(2, 1 -1) 

= E r' T R(u + ^)^ lT du. (48d) 

Assuming that 

Z r (« + *£) <r' r 

is a uniformly convergent series 

R(u>) I — » — , <?^2, as o> — » «> 

U) q 

r, - f' T E R(u + 2 -^)e iulT du. (48e) 



(' 
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Noting that n is just the Ith. coefficient of an exponential Fourier series 
expansion of 

27T -^ „ / . 2tt7r\ 7T 7T 

U + -=r- J — ?p ^ U ^ ■=, 



or 



and 



T JL fl V" ' t ) t= - = r 

one may write 

JL B ( M + t) = £ ,£. "•""*" (49a) 

t R (u + ^) - p = f £ W-*" (49b) 

\t r(u + 2 ^)- ^l* e j » mT = ^Z r^ lT ^ mT . (49c) 

Multiplying (49b) and (49c) together and integrating one obtains, 
using the shorthand notation 

«.(«) = R (u + 2 -^) , 

2 f* IT I r T I 2 

Zl '•!»'!+». = -Z / 2Z ^ n (w) - tJ- cos umT du. (50) 

2^0,-m ■/ J-r/T | n « | 

Further, 

00 

fl/E''| 2 + 2j; a^+n, Z) ?*^i+m 

(.51) 

where 

F(u) = a/ + 2 2 a^, cos MmT. (5) 

m— 1 

The noise variance may be written 

E\nJ) = [ \E(to)\ 2 <$>(u>) du> (52a) 

where <J>(co) is the channel noise power spectral density 

E{nf\ = E ["' ' \E(a>)\ 2 *Mdu (52b) 

n J-r/T(.2n— 1) 
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E(u + -= 



n «*— 

= f /r EiJ.(«)r*b(i*) 

J—t/T n 



:)| f *(u + ^du (52c) 



du 



(52d) 



if E« | l?„(w)| 2 $„(m) is a uniformly convergent series. The mean 
square error expression may then be written 

M.S.E. = f T {e I E n (u)\ 2 *,.(«) 

*»-T/r n 

(53) 



+ Y 



[Re £«.(*) -^]V(w) +~\lmZRn(u) 
n 2jtJ i L 



F(u))du. 



In terms of the system component functions 

S(a>) = A(«)e ia(w> 
T(«) = B(o)J' w 
E(u) = C(u>)e' y(u) 

(53) may be written 

M 



.&E7. = f JE C„ 2 (w)#„(m) 

XI A(ii)B n («)C»(tt) cos [a„(w) + /3 n (w) 



+ 



2tt 



+ 7.(t*)] - tP *(«) 



roTT 

27TJ 



(4) 






T A n (u)B n {u)C n {u) sin [a n (w) + /3„(tt) 



+ y a (u)] 



F(u) } du. 



APPENDIX B 

Simultaneous Reduction of Noise and Intersymbol Interference 
The function 

r" \Z C n % n + *%■ (E ^C cos («. + ft. + 7») - "Pi 
+ ^ JE A n B n C n sin («. + /3 n + 7n)l J du 
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+ Xi / FEi„ 2 du 

J-t/T n 

pT/T 

+ X 2 / J2 A n B n C„ cos (a„ + ft + y„)du 

J-r/T n 

is to be minimized subject to the constraints 

['" FT,A n 2 du = P (9) 

J-r/T 

and 

I £ A n B n C n cos (a„ + ft + 7„)dw = r . (8) 

J-r/T n 

For the optimum signal, this minimization is carried out with respect 
to Ak and a* . Using the standard techniques of the calculus of varia- 
tions 9 the minimization with respect to ak yields 

A k B k C k sin (a k + ft + 7*) 2 A n B n Cn cos (<x„ + ft + 7») 

_ I. f 2 r - ^1 - 4M cos U + ft + 7*) (54) 

•|~£ A n J5„C„ sin (o» + A. + 7,)] = 0, 

assuming that F 5* 0. The special case when F = will be considered 
later. Summing over all k leads to 



(2ro - y) S ^*»*C* s^ (a* + ft + 7*) = 



(55) 



since X 2 depends upon the constraints 

£ A^C* sin (a* + ft + 7*) = (56) 



and 



A k B k C k sin (a k + ft + 7*) 2 A n B n C n cos (a n + ft + 7*) 



-5l<*»-? 



(57) 



= 



Minimizing with respect to Ah one gets 
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p- F[BkC k cos U + fa + 7*)] 



E A n B n C n cos (a„ + 0„ 



+ 7, 



+ — F[B k C k sin (a fc + (3 k + 7*)] , s 

r (58) 



Kn) " 2VJ 

•|~E ^ n -B„C n sin (a n + /3 n + 7n)J 

+ 2Xt FA k + X 2 B*C* cos (a* + 0* + 7*) = 
or using (56) 

[B k C k cos (a t + 0* + 7*)] E ^n^»C„ cos (a» + /3» + 7..) 

r /_ X 2 \l , 2\ X T 



_T 

47 

Comparing (57) and (59) it is seen that a nontrivial solution requires 

sin (a* + /3a- + 7*) = (10) 

and 

Ak = - * B k C k JE ii,B,C„- ^ (2r - *?)} . (19) 

Multiplying each side of the equation by B k C k and summing one gets 

E A k B k C k = -^ E B k 2 C k > JE AAC. - T - (2r - ^)} . (60) 



E ilftftC* = £; 

t 4tt 



BkCk 



k ait 



(61) 



and 



?«-£(*-*)--£($ 



(2r - X 2 /F) 

k 4TT 



Finally, 






• (62) 



(11) 
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For the optimum receiver filter, the minimization with respect to 
7^ again yields (56) and (57) and with respect to C k leads to 

— F[A k B k cos (a k + fi k + 7*)] E A n B n C„ cos (a„ + /3„ + y n ) 

1 _ n 

" 27"] + ¥ F[AkBk sin {ak + ft + 7k)] (63) 

[~E A n B n C n sin ( a „ + /3 n + 7»)] 

+ 2Ct$ft + XoAftBft cos (at + /3 A . + 7*) =0 
or using (56) 

[A k B k COS (a k + /3ft + 7*)1 E AnBnCn COS (a n + ft, + 7n) 



-£(»-*)]+*» - 



(64) 



4tt F 

Again, comparing this equation and (57) reveals that a nontrivial solu- 
tion requires 

sin (aft + /3ft + 7a-) = 0. (10) 

Then 



0* = -=?■ 



2tt ^ftBftF 



EAAC„-^(2ro-^)]. 



Multiplying each side of the equation by A k B k and summing one gets 



(20) 



E Wft =-Y?Z^[i: A^Cn - £ ( 2 r - *)] (65) 






E^BftCft-- ^ y 

and 



^,,.-t { ,.-^.-ua r ^ 



k $ft 2x 



(67) 



Finally, 
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(12a) 



The coefficient, T/4ir(2r — X 2 // r ) may be determined from the con- 
straint 



/ JlA k B k Ck du = r ■ 

J— ir/T k 



(8) 



Multiplying both sides of (12a) by A k B k , summing, and integrating 
one obtains 



To 



4t J- tIT y ° F 



„-sp A k B k 

k $k 

k <Pfc 2lT 



(III 



or 



(-» 



2x 



jt/7' Z^ 

* "Pa- 



/T/7' 
-ir/r 



aw 



^ 2 

/ "Pa- 



'-«•/!■ ny A k ~B k ~ , 7 1 



i r /r 

F J-T/T 



— du 



■it ^ A k 2 B k 2 



fZ 



<p k 



+ 



2ttJ 



T/T 



£ 



«p fc 



k "P* 27T 



di* 



The mean square error 

r/T 



/ 2 

J —r/T L * 

may be found using (12a,b) to be 



C^+^lZAkB^-f 

IT L * ■*■ I * ^T 



dw 



2tt 



A.-fii 






J—tc/T A •'— t/5 



,T_(\ TlT du 
2tt V-,/1. A 



i/r y^ A k B k 



-T/T 



k <p/,- 



du 



(68) 



(12b) 



where 



2392 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1965 



REFERENCES 

1. Nyquist, H., Certain Topics in Telegraph Transmission Theory, AIEE Trans., 

47, April, 1928, pp. 617-644. 

2. Bennett, W. R. and Davey, J. R., Data Transmission, McGraw-Hill Book Co., 

New York, 1965. 

3. Tufts, D. W., Matched Filters and Intersymbol Interference, Cruft Laboratory, 

Harvard University, Cambridge, Massachusetts, Tech. Rept. No. 345, July, 
1961. 

4. Tufts, D. W., Nyquist's Problem in Pulse Transmission Theory, Cruft Lab- 

oratory, Harvard University, Cambridge, Massachusetts, Tech. Rept. No. 
425, Sept., 1963. 

5. Tufts, D. W., Certain Results in Pulse Transmission Theory, Cruft Labora- 

tory, Harvard University, Cambridge, Massachusetts, Tech. Rept. No. 335, 
Feb., 1962. 

6. Tufts, D. W., Nyquist's Problem — The Joint Optimization of Transmitter 

and Receiver in Pulse Amplitude Modulation, Proc. IEEE, 63, March, 1965, 
pp. 248-259. 

7. Gibby, R. A. and Smith, J. W., Some Extensions of Nyquist's Telegraph Trans- 

mission Theory, B.S.T.J., U, Sept., 1965, pp. 1487-1510. 

8. Bennett, W. R., Statistics of Regenerative Digital Transmission, B.S.T.J., 37, 

Nov., 1958, pp. 1501-1542. 

9. Courant, R. and Hilbert, D., Methods of Mathematical Physics, 1, Inter- 

science Publishers, New York, 1953. 

10. Lender, A., The Duobinary Technique For High Speed Data Transmission, 

IEEE Trans, on Communication and Electronics, 82, May, 1963, pp. 214-218. 

11. Bennett, W. R., unpublished memorandum. 



